Performance Evaluation with Industrial Applications

It is being increasingly understood that the evaluation of computer systems performance must be based on realistic, full applications. Kernel, algorithm, and small-application benchmarks are important for measuring and discussing the performance of individual components of a computer system. However, the complex performance behavior of these components in concert and in the context of a large computational problem can only be realistically understood if we observe the problem as a whole. These needs are further increased by the fact that computer systems are able to solve ever larger and more complex problems. Doing performance evaluation at this large scope is a grand challenge and exceeds the resources of most research projects. To address this challenge, a joint industrial-academic e ort has been initiated with the mission to facilitate performance evaluation and benchmarking with large, industrial applications. There are two complementary goals. The industrial participants pursue the development of systems benchmarks that provide meaningful and relevant applications for measuring and comparing high-performance computer systems across diverse platforms. The academic participants seek realistic program suites for evaluating and guiding the research of new concepts and prototypes. Both performance evaluation e orts have the need for large-scope, industrially relevant applications.